Introduction to virtual reality

An interactive world, a moving body, a perceptual experience

Étienne Peillard
IMT Atlantique

The eight-course roadmap

Understand the field → explain perception → design systems → evaluate experience

Outline of the introduction course

Definitions

1.1

Action and perception

1.2

Quiz: VR, AR or neither?

1.3

1.1

Definitions

Virtual reality

Replace the real world with a virtual world through perception and action interfaces.

Virtual reality

In the physical-world situation on the left, perception and action directly relate the person to the environment. The user perceives the world through sensory information and acts on it through movement. Their actions change the world or their position within it, which changes what they perceive. This is the perception–action loop.

The VR situation on the right preserves this loop while placing technical interfaces between the user and a virtual world. Sensory interfaces generate the stimuli through which the user perceives that world: images, sounds and, when available, haptic feedback. The perception arrow therefore runs from the virtual world, through the sensory interfaces, to the user.

In the opposite direction, tracking systems, controllers or other input devices capture the user’s actions and translate them into actions in the virtual world. The simulation updates the environment and produces the next sensory feedback. The action arrow therefore runs from the user, through action-capture interfaces, to the virtual world.

The defining idea is the mediated perception–action loop: a virtual world takes the place of the physical world in the interaction, with interfaces providing the sensory information and the means to act. An artificial environment alone does not explain this relationship. The environment can represent a real place, make an otherwise invisible phenomenon perceptible, or create an imaginary world; the interfaces maintain the coupling in real time.

Augmented reality

Keep the physical world

Objects, surfaces and physical constraints

Keep its spatial structure

Positions, distances and reference points

Add virtual elements

Objects and information situated in the scene

Information overlaid on the physical environment

Keep the physical world as the setting of the experience; add virtual elements within it.

Augmented reality

The starting point of augmented reality is the real environment. Its objects, surfaces, material properties and physical constraints remain part of the experience. The user continues to inhabit and act in that environment.

AR preserves the environment’s spatial organisation: the positions of objects, distances, surfaces and reference points remain meaningful. Virtual elements are added in relation to that setting, for example an object on a real table, guidance attached to a machine, or information associated with a place. Spatial alignment helps the added content remain coherent with the real scene.

The design objective is to enrich the real environment while retaining its physicality and spatiality. Optical see-through, video see-through, handheld screens and projection are different ways of pursuing this objective. The experience’s relationship with the real environment is the key point of this definition.

VR and AR: two design goals

Different goals; devices and perception–action mechanisms can be shared.

VR and AR: two design goals

The distinction concerns the objectives of the experience. In VR, the goal is to substitute a virtual environment for the real environment in the user’s perception–action loop. In AR, the goal is to retain the real environment and enrich it with virtual elements.

Both diagrams begin with the same situation: a physical room with a table. On the left, the room is replaced by a simulated setting. On the right, the room and table remain, with a virtual object added in their spatial frame. Grey represents the physical environment; cyan represents virtual elements.

These objectives do not map to a simple hardware division. Tracking, displays, rendering and interaction techniques can be shared, and a device can support different experiences. A passthrough headset, for example, can retain a view of the real scene and add virtual objects, while also supporting a virtual experience that replaces the visible surroundings.

The two approaches also share the perception–action framework: the user perceives and acts, and the system responds in real time. The distinction emphasised here concerns the role assigned to the real environment and the intended experience. The following continuum describes how real and virtual elements can be combined.

The historical continuum

Milgram and Kishino · 1994

The view can be virtual: what remains real in the experience?

Real content inside a virtual world

Augmented virtuality · two examples

Spatial · a webcam inside the VR room

Spatial screenshot: the “Bri Scully’s Webcam” video feed appears beside avatars in a virtual room.

A participant’s camera feed appears inside the virtual room alongside avatars.

Spatial · screenshot published by WIRED · 2020 ↗

Quest 2 · Space Sense (Guardian)

Quest 2 capture: pink outlines of real objects are superimposed on the virtual environment.

Outlines of real objects and people are revealed within the play area, over the VR environment.

▶ In-headset demonstration · Skarredghost · 2021 (2:33) ↗

The setting is virtual; visual information from the real world is brought into it.

Real content inside a virtual world

The collaborative version of Spatial shown in 2020 allowed participants without a headset to share a webcam inside a virtual room. In this screenshot supplied by Spatial and published by WIRED, the “Bri Scully’s Webcam” panel is a real camera feed. Other people appear as avatars: distinguish the video panel from the textured avatar faces.

The second capture comes from Antony Vitillo’s (Skarredghost) hands-on test of Space Sense on Quest 2 in November 2021. The linked video starts at 2:33, at the beginning of the in-headset demonstration. Space Sense overlays camera-derived outlines of real objects and people onto the VR view. This is more than the virtual Guardian boundary grid, and is not a full camera view of the real world.

Space Sense was introduced as an experimental Roomscale Guardian feature on Quest 2 with v34 in 2021. This capture documents that historical behaviour rather than its current availability on all headsets.

This slide is a vertical detail beneath the continuum: use the down arrow to visit it, and the up arrow to go back.

The continuum revisited

Body, support, movement limits · Skarbez, Smith and Whitton, 2021

The real body keeps conventional virtual reality within mixed reality.

Interaction paradigms

Graphical interfaces

How do we act on screen objects?

Ubiquitous computing

How do we coordinate distributed systems?

Immersive interfaces

How do we act in a simulated world?

Augmented interfaces

How do we link additions to the real scene?

Four illustrated scenes: a desktop graphical interface, connected everyday objects, an immersive VR headset and controllers, and AR glasses overlaying a real engine.

Google Glass archive

Observe: does information follow the head or remain linked to the scene?

Information in the field of view is not necessarily attached to the world.

HoloLens archive

Observe: which movement reveals how objects are anchored in the room?

Look for the spatial link; a promotional video does not describe the full field of view.

ARKit demonstration

Observe: does the object stay in place as the tablet moves?

The shared problem: maintain correspondence between real and virtual.

1.2

Action and perception

One gesture, one loop: grasping an object

What is measured? What changes? What feedback is produced?

Action: person → world. Sensory feedback: world → person.

Headsets and projection rooms

Choose for the task: individual viewing, sharing, collaboration

Headset

Individual view; share through an external screen

Projection room

Side-by-side observation; tracking determines perspective

Comfort, collaboration and infrastructure: the activity guides the choice.

1.3

VR, AR or neither?

Street View

VR, AR or neither?

Interactive panorama

360° photos can be explored, without tracking body movements.

AirPano · Maldives

VR, AR or neither?

Immersive video, limited interaction

We choose where to look; the flight remains prerecorded.

Behind the wheel of a Toyota

VR, AR or neither?

Head-up display, not spatial AR

Information stays within a display area; it is not anchored to objects on the road.

Benedict Cumberbatch · Smaug

VR, AR or neither?

Visual effects

The performance helps create the dragon; the film does not respond to the viewer.

F1 · 4DX cinema

VR, AR or neither?

Multisensory cinema

The effects follow the film, not the viewer’s actions.

Hello, Dot

VR, AR or neither?

AR

The virtual companion occupies the real living room and responds to gestures.

Snapchat · face filter

VR, AR or neither?

AR

The virtual effect follows the face in real time.

Ghost Town

VR, AR or neither?

VR

Movements and gestures allow interaction with the simulated world.

CAVE · University of Bacău

VR, AR or neither?

The University of Bacău laboratory uses a projection cube and simulation software for interactive activities.

Room-scale VR

A simulated environment can surround the user without a headset.

Takeaways

Act and perceive within a simulated world

Enrich reality with virtual content registered in real time

A headset and rich sensations alone are not enough.