Perception and theoretical foundations

The mechanisms behind immersive experience

Étienne Peillard
IMT Atlantique

Course outline

Definitions and models

4.1

Spatial perception

4.2

Multisensory perception

4.3

Sensorimotor coupling

4.4

Perceptual illusions

4.5

4.1

Definitions and models

“Virtual reality”: what do the words mean?

An English expression, several readings in French

Virtual · English

In effect · almost · functioning as such

Virtuel · philosophy

Potential rather than actual — also a French meaning

An apparent oxymoron

If virtuel is read as unreal, the French expression réalité virtuelle sounds contradictory

For VR: a simulated environment that functions as a reality for perception and action.

“Virtual reality”: what do the words mean?

The term cannot be understood only by opposing real and unreal. In ordinary English, virtual can describe something that is almost so or functions as such without having exactly that status. The computing sense also refers to a software-based simulation.

The historical root evokes force or power. However, potentiality is not uniquely English: French virtuel also has a philosophical meaning opposed to actual. It would be misleading to say that English virtual simply means potential while French virtuel only means unreal.

The French expression can sound like an oxymoron when virtuel is read as unreal. The paradox depends on the chosen meaning, rather than being a necessary contradiction in the technology. The final formulation connects to our approach: simulating perception and action in relation to an environment.

A name that was not self-evident

Several terms coexisted, each emphasising a different aspect.

Réalité artificielle

Artificial reality: a constructed environment

Réalité vicariante

A proposed French alternative: a substitute that functions as reality

Réalité de synthèse

Computer synthesis: the French official term published in 2007

Réalité virtuelle became the established French usage, despite terminological reservations.

A name that was not self-evident

The vocabulary was not settled in a single step. These competing proposals and usages show that the name was not universally accepted. The three cards compare conceptual emphases rather than presenting a strict chronological sequence.

Philippe Fuchs discusses the proposed expression réalité vicariante. French vicariant means something that replaces or substitutes for something else. It highlights functional substitution, rather than mere potential existence or the need for a visually identical copy of the physical world.

FranceTerme records réalité de synthèse as an official French term published on 20 April 2007. This recommendation does not mean that it displaced réalité virtuelle in ordinary usage. The December 2022 ÉduNum publication notes that réalité virtuelle became the established expression despite alternative proposals.

Several definitions, several viewpoints

Purpose

The human activity to support

Function

Changing place, time or possible action

Technique

Simulation and behavioural interfaces

Interface

How the person relates to the world

Relationship with reality

What simulation makes us notice

Perception

How the user interprets the experience

Defining virtual reality by its purpose

The purpose of virtual reality is to enable a person (or several people) to carry out sensorimotor and cognitive activity in an artificial, digitally created world, which may be imaginary, symbolic, or a simulation of certain aspects of the real world.

Burkhardt and Fuchs · 2006

ViewpointHuman activity

A functional definition

Virtual reality will allow the user to extract himself from the physical reality in order to virtually change the time, place and/or type of interaction: interaction with an environment simulating reality or interaction with an imaginary or symbolic world.

Fuchs, Moreau and Guitton · 2011

ViewpointChanging the frame of possible activity

A technical definition

Virtual Reality is a technical and scientific area making use of computer science and behavioral interfaces in order to simulate 3D entities behavior in a virtual world that interact in real time among themselves and with the user in pseudonatural immersion through sensory-motor channels.

Fuchs, Moreau and Guitton · 2011

ViewpointSimulation, interfaces and real time

Defining the interface relationship

The substitution of the interface between a person and the physical environment with an interface to a simulated environment.

Jaron Lanier

ViewpointMediating the person–world relationship

Defining a relationship with reality

A coarser, simulated reality fosters appreciation of the depth of physical reality in comparison. As VR progresses in the future, human perception will be nurtured by it and will learn to find ever more depth in physical reality.

Jaron Lanier

ViewpointReflecting on the experience of reality

Defining the role of perception

Technology that rallies the brain to fill in the blanks and cover over the mistakes of a simulator, in order to make a simulated reality seem better than it ought to.

Jaron Lanier

ViewpointThe user contributes to the experience

Virtual reality and LSD: a shared world

… VR users can share a world objectively, even if it is fantastical, while LSD users cannot.

Jaron Lanier · Dawn of the New Everything (2017)

ViewpointExperiencing a common world

The I²: immersion and interaction

I² denotes the immersion–interaction pair

Immersion: perceiving the virtual world

The device engages the senses and lets the user appropriate a new perceptual space. Example: turning the head to explore the scene.

Interaction: acting in the virtual world

The user acts and perceives the consequences of their actions in real time. Example: grasping a part and seeing it follow the hand’s movement.

In 3I², this pair applies at three levels: sensorimotor, cognitive and functional.

The I²: immersion and interaction

The two I’s stand for immersion and interaction. The notation I² associates these two closely related concepts. The number 3 denotes the sensorimotor, cognitive and functional levels, detailed on the next slide.

In Auvray and Fuchs’s approach, perception is an activity: users discover regularities between their actions and sensory feedback. Immersion therefore also depends on how they appropriate the device to perceive and act. It involves more than wearing a headset or receiving realistic images.

Turning the head to explore a scene already involves an action and its perceptual feedback. Grasping a part illustrates the same coupling: the hand’s movement controls the object’s movement, which the user can see. The examples distinguish the concepts for teaching purposes while showing their interdependence.

Reference: M. Auvray and P. Fuchs (2007), Perception, immersion et interactions sensorimotrices en environnement virtuel, Intellectica 45, pp. 23–35, sections 1 and 2. https://intellectica.org/SiteArchives/archives/n45/45-1-Auvray.pdf

The 3I² reference model

3 levels × (Immersion + Interaction)

Design from the activity; check that all three levels work together.

The 3I² reference model

3I² means three levels of immersion AND interaction: sensorimotor, cognitive and functional. It is a reference framework for designing and evaluating virtual-reality applications, rather than a score of immersion. The two columns relate the user’s activity to the technical construction of the virtual world.

At the sensorimotor level, senses and motor responses are coupled to the simulated world through sensory and motor interfaces, their drivers and real-time computation. Relevant questions include tracking, latency, sensory resolution and consistency between action and feedback.

At the cognitive level, the user appropriates a way of perceiving and acting. An acquired action schema can be transferred from everyday life, or an interaction metaphor can be learned. On the system side, behavioural models and software aids support these actions. A familiar gesture alone does not guarantee that the interaction will be understood or usable.

At the functional level, the question is which activity the application must enable and for what purpose. Activities can be decomposed into observing, moving, acting on the virtual world and communicating. Start from the required activity before choosing interfaces. The bidirectional links indicate dependencies between levels, not three successive stages or three independent qualities.

3I² in practice: grasping a part

One assembly task, three linked design questions

Too much latency → difficult grasping → the assembly task is compromised.

3I² in practice: grasping a part

This teaching example applies the model to a virtual assembly exercise. At the functional level, the learner must grasp a part and place it at a target. The purpose and success criterion come before the choice of device: which placements, constraints and tolerances matter for the learning objective?

At the cognitive level, the interaction uses a familiar grasp-and-place schema, adapted to a tracked pinch. The application must implement understandable rules for attachment, movement and release. Snapping near the target is an example of a behavioural software aid; whether it is appropriate depends on what the learner is meant to practise.

At the sensorimotor level, hand tracking captures movement and pinch while the headset provides visual feedback. Low latency and stable tracking help the user experience the object as moving with the hand. Delayed feedback can make grasping unreliable, disrupt the action schema and prevent completion of the assembly task. Conversely, responsive hardware does not establish that the learning objective has been achieved.

Fidelity depends on the task

Physical fidelity

How stimulation matches the target situation

Functional fidelity

Whether the relevant actions are possible

Psychological fidelity

Whether the relevant responses occur

4.2

Spatial perception

Spatial perception: the route through this section

Which information helps us perceive depth, distance and scale?

1. Monocular image cues

Occlusion · perspective · texture · size and height · shadows

2. Binocular disparity

Differences between the two eyes’ views

3. Motion parallax

Relative image motion as the viewpoint moves

4. Vergence and accommodation

Eye alignment and optical focus

5. Perceived distance and scale

Putting the cues in context: viewpoint height, familiar size and field of view

Spatial perception: the route through this section

This overview follows the order of the section. We first examine cues available in a single image, then binocular disparity, then the relative image motion produced by moving the viewpoint. The head-tracking demonstration illustrates this third family.

We then distinguish vergence from accommodation, using the focus video as an analogy for optical focus. These are different mechanisms, even though they are usually linked in natural viewing.

The final slide returns to the outcome: perceived distance and scale. Viewpoint height, familiar size and the available field of view provide context; this final step is not an additional, independent family of depth cues. The families overlap: motion parallax can be perceived with one eye, and familiar size already appears among monocular cues.

Monocular depth cues

Occlusion

Which surface hides another?

Perspective and texture

How patterns change with depth

Size and height

Objects relative to familiar references

Shadows

Evidence about contact and position

Binocular disparity

Two viewpoints

The eyes occupy different positions

Two projections

A feature falls at different retinal positions

Depth interpretation

Disparity contributes to perceived distance

Motion parallax

Two fixed objects · head translation without rotation

In VR, rendering must follow head position to reproduce this depth cue.

Motion parallax

Motion parallax provides a depth cue when the observer translates. The diagram considers two fixed objects and a rightward head translation without rotation: for the same head displacement, the near object has a larger apparent shift than the far object. The objects do not move in the world; their directions relative to the observer change.

On the left, lines connect each head position to the two objects in a top view. On the right, both frames use the same viewing orientation and projection scale. The blue circle is near and the brown triangle is far. From the first viewpoint to the second, both shift left, but the nearer object shifts farther. The two lower arrows compare the displacements.

This depth cue is available to one eye. The illustration assumes constant viewing orientation; visually tracking an object can alter the directions of relative retinal motion while preserving the difference between depths.

To reproduce this effect in VR, the system must track head position and recompute the viewpoint. Rotation tracking alone is insufficient. With a fixed display, this involves adapting projection to eye position, as illustrated in the following video.

The viewpoint follows head movements

Johnny Lee · 21 December 2007

Vergence and accommodation

Vergence

Aligning both eyes on a target

Accommodation

Focusing the eye’s optics

Conventional stereo displays

These demands can differ

Perceived distance and scale

Viewpoint height

A reference for the scene

Familiar size

A cue to object dimensions

Available field of view

Changes the visible spatial context

4.3

Multisensory perception

Sensory channels and information

Exteroception

Signals about the surrounding world

Proprioception

Position and movement of the body

Interoception

Signals from inside the body

Vestibular information

Head motion and orientation to gravity

Spatial hearing

Localisation

Direction and distance cues

Material and event

What happened, and to what?

Attention

A signal outside the current view

Touch and active exploration

Cutaneous information

Pressure, vibration and texture

Kinesthetic information

Movement, forces and resistance

Active exploration

Touch combined with movement

Contact with an actively explored virtual surface

Whole-body feedback: a moving walking surface

Whole-body feedback: the Virtusphere

Whole-body feedback: a movable support surface

The vestibular system

Semicircular canals

Angular acceleration

Otolith organs

Linear acceleration and gravity

Integrated estimate

Movement and balance

Smell and taste

Smell

Identity, memory and context

Taste

Chemosensory and multisensory experience

Practical constraint

Delivery and persistence

Multisensory taste demonstration

Multisensory integration

Redundancy

Several cues indicate the same event

Complementarity

Different cues add different information

Congruence

Timing and spatial alignment support a common cause

4.4

Sensorimotor coupling

Sensorimotor contingencies

An action

The head turns or a hand reaches

An expected change

The view or hand should update

A perceived result

Feedback confirms or violates the expectation

Spatial and temporal coherence

Space

Inputs and outputs share an alignment

Time

Feedback follows the action with limited delay

Modalities

Events agree across sensory channels

Physical and virtual surroundings

Physical support

Props or equipment serve the task

Neutral background

Some real cues remain irrelevant

Interference

Real cues contradict the virtual situation

4.5

Perceptual illusions

Pseudo-haptic feedback

Physical input

A movement or force

Visual manipulation

Change displacement or deformation

Haptic interpretation

An apparent resistance or material property

Visual manipulation of apparent friction

Try it: feel resistance through vision

Palpation: changing visible deformation

Control–display gain

Gain = virtual displacement / physical displacement

Physical displacement

The movement made by the person

Virtual displacement

The movement shown by the system

Gain

The relation between the two

Pseudo-haptics in action

Apparent resistance

Alter displayed motion to suggest resistance or surface relief.

Apparent impact

Briefly stop or slow the virtual racket at contact.

Pseudo-haptics in action

On the left, Kenichi Ito’s WebXR demonstrations change the mapping between input and displayed motion to illustrate resistance and surface-relief sensations. Observe the mismatch between the intended movement and the visible response. The video presentation is in Japanese; the animations can be explained orally.

On the right, Hit-Stop in VR temporarily changes the virtual racket’s motion when it contacts the ball. The study combines this visual effect with vibration, so the entire impact sensation should not be attributed to visual feedback alone. The demonstration allows comparisons of stopping versus slowing, duration, and the presence or absence of vibration.

These videos show visual-feedback manipulations, not the viewer’s subjective sensation. Experiencing pseudo-haptics requires acting oneself and perceiving the mismatch between action and displayed outcome.

Perceptual flexibility

Apparent material

Pseudo-haptic contact

Apparent path

Redirected walking

Apparent body

Ownership illusions

Explaining an illusion

Which signal changes?

Which cues remain consistent?

What does the participant infer?

When might the effect fail?

Sources and further reading

Teaching resources

Original lectures and their figure credits

Research references

Study-specific citations in each slide’s notes

Source map

References retained at the point of use

The Virtual Reality Treatise All course materials