Building a VR system

Motion capture and sensory output

Étienne Peillard
IMT Atlantique

Course outline

Motion capture

5.1

Sensory output

5.2

5.1

Motion capture

Six degrees of freedom

Translation

Left/right, up/down, forward/back

Rotation

Yaw, pitch and roll

Mechanical tracking

Telepresence Mobile Robot (1991)

SPIDAR-8

Electromagnetic tracking

Polhemus FASTRAK system with transmitter and wired sensor — Polhemus FASTRAK

Current tracking technologies

Participant wearing optical markers in an OptiTrack capture room for the MuscleRehab project — Optical: OptiTrack (MuscleRehab)

Two people wearing Xsens Link suits with sensors on body segments — Inertial: Xsens Link suit

Current tracking technologies

Optical tracking uses visible or infrared observations. Depending on the device, it tracks known markers, environmental features or light signals. Cameras may be external or onboard. The next slide distinguishes inside-out and outside-in arrangements.

An inertial measurement unit (IMU) typically contains gyroscopes and accelerometers. Gyroscopes measure angular velocity. Accelerometers measure specific force, which includes the effect of gravity depending on sensor orientation. Estimating displacement requires accounting for this component and integrating measurements. Bias and error accumulate: an IMU alone does not provide drift-free absolute position.

Inertial measurements enable rapid updates between optical observations. Optical observations provide constraints that limit drift. Sensor fusion is discussed after the tracking arrangements.

Documented example: Microsoft describes headset tracking that fuses camera observations with high-rate IMU measurements. https://learn.microsoft.com/en-us/windows/mixed-reality/enthusiast-guide/tracking-system

Left, the MuscleRehab setup uses OptiTrack cameras and body-worn markers for optical motion capture. Image annotations identify cameras and markers. The project’s EIT muscle measurements are separate from optical tracking. Right, the Xsens Link suit illustrates inertial tracking using body-worn sensors. Photo sources: MIT CSAIL, https://hcie.csail.mit.edu/research/musclerehab/musclerehab.html and Xsens, https://www.xsens.com/motion-capture/xsens-link-specifications

Inside-out and outside-in: principle

Two comparative diagrams: left, cameras fixed in the room observe the headset; right, headset-mounted cameras observe the environment. — Outside-in: external cameras. Inside-out: onboard cameras.

Inside-out and outside-in tracking

Arrangement	Observation	Typical limitation
Inside-out	Sensors carried by the tracked device	Needs usable environmental observations
Outside-in	External sensors observe tracked objects	Needs coverage and line of sight

The Quest’s four onboard cameras, circled in purple — Inside-out: cameras on the headset (Quest)

External Rift CV1 tracking camera on its stand — Outside-in: camera in the room (Rift CV1)

Sensor fusion

Fast local changes

Inertial measurements

External reference

Optical or other observations

Combined estimate

Pose with reduced drift

Camera images and inertial measurements converge in a sensor fusion module that estimates pose. — Camera–IMU fusion example

Sensor fusion: OKVIS

Tracking quality

Accuracy

Closeness to the reference pose

Jitter

Rapid variation around an estimate

Drift

Accumulating displacement over time

Loss and recovery

Behaviour when observations disappear

Controllers

Half-Life: Alyx

Close-up of a Valve Index controller with buttons and grip — Valve Index

Hand tracking

Silhouette · Ultraleap

Leap Motion sensor and hand joint visualisation on a screen — Leap Motion

Hand tracking

Hand tracking reconstructs an articulated skeleton from observations. A joint estimate can become uncertain when another finger, the other hand or an object hides it. Hands must remain within the cameras’ observable region.

A pinch is one possible command: bringing the thumb and index finger together can trigger selection or grasping. Distinguish the estimated hand pose, gesture recognition and the action assigned by the application.

The illustration shows a Leap Motion sensor and estimated joints on screen. The Silhouette video shows interaction through hands and their shadows, as presented in Ultraleap’s documentation. Optical tracking alone provides no mechanical resistance.

Illustration: https://documentation.ls-group.fr/interact/LeapMotion/ Demonstration: https://docs.ultraleap.com/ultralab/vr-gaming-controllers-and-hands.html

Eye tracking

Tobii VR Integration Demo

Eye tracking

Head-gaze uses headset orientation. It approximates the direction being looked at: the eyes can move without the head moving. Eye tracking uses dedicated sensors to estimate gaze direction.

Eye tracking may require user-specific calibration. Accuracy and data availability depend on the device and operating conditions. A gaze direction is neither proof of attention nor an explicit selection intent.

Gaze-and-commit separates targeting from confirmation. A button, pinch gesture or dwell time can confirm an action. Separate confirmation lets users visually explore the scene without every glance triggering an action.

References: Microsoft Learn, Eye tracking overview, https://learn.microsoft.com/en-us/windows/mixed-reality/design/eye-tracking and Eye-gaze-based interaction, https://learn.microsoft.com/en-us/windows/mixed-reality/design/eye-gaze-interaction

Illustration: https://www.tobii.com/resource-center/learn-articles/how-do-eye-trackers-work Demonstration: https://video.tobii.com/tobii-vr-integration-demo

Body tracking

Rokoko · Smartsuit Pro II

Body tracking

Body tracking can use cameras, optical markers or wearable sensors. The number and placement of observations determine which segments have supporting measurements. Not every segment visible on an avatar is necessarily measured.

Inverse kinematics can infer a posture compatible with tracked points and a skeleton model. If only the head and hands are tracked, several elbow and body configurations can fit the same observations. Reconstruction therefore requires constraints and assumptions.

A mismatch between real movement and avatar movement can disrupt coordination and bodily experience. Distinguish available measurements from reconstructed segments and match avatar detail to tracking quality. The avatar animation section returns to this mapping.

Illustration and video: Rokoko Smartsuit Pro II. The demonstration compares the recorded performer, the skeleton in Rokoko Studio and the character in Unreal Engine. Source: https://www.rokoko.com/products/smartsuit-pro

5.2

Sensory output

Visual stimulation

Present a scene, its depth and changes in viewpoint

Families of visual displays

Head-mounted displays

An image displayed in front of the eyes

Projection and screens

A physical surface shared by or surrounding users

Augmented reality

The physical world remains visible with added content

Head-mounted display: FlyVIZ

Immersive projection: a CAVE

Projection systems need calibrated geometry and viewpoint-dependent rendering. Several people can occupy the room, but a shared stereo image pair is geometrically correct only for the intended tracked viewpoint. Multi-view systems can address some constraints at additional complexity. The visible physical body and access to props can be useful.

A headset and a projection room can both support immersive experiences. Their constraints differ in field of view, optical properties, mobility, visibility of the physical body and collaboration. Tracked projection must generate an image appropriate to the observer’s position. Device selection follows the task, rather than a single hierarchy of immersion.

Correct stereo rendering requires more than offsetting two images. The eye locations and display geometry determine the projections. A CAVE or desktop display needs off-axis rendering as the observer moves. The two eye views should remain consistent with world scale and the participant’s calibrated interocular distance.

Remote collaboration depends on which information about others becomes available. Fleury and colleagues’ simplified avatars communicate position and action in a shared environment. Beck and colleagues’ group telepresence uses three-dimensional video-related representations and collaboration tools. These arrangements make different aspects of posture, expression and shared activity visible; appearance and coordination are related but distinct design questions.

Active stereo: shutter glasses

Active shutter glasses — Alternating left and right views, synchronised with the glasses

Passive stereo: polarisation

Polarisation-based separation — Two polarisations, one view for each eye

Anaglyph stereo

Anaglyph separation — Views separated by colour filters

Autostereoscopic displays

An autostereoscopic example — Views directed to each eye, without glasses

Multi-user projection: C1x6

Augmented reality techniques

Optical see-through

Direct optical view

Real light reaches the eye through a transparent combiner; a projector adds virtual imagery to the optical path.

Real light + optically added virtual content

Video see-through

Camera-mediated view

Cameras capture the real scene. A processing stage combines it with virtual graphics before showing the result on an opaque display.

Camera → composition → display → eye

Handheld display

Phone or tablet

Hands hold a tablet whose camera captures the real setting; its screen shows the setting with a virtual object added, while the surroundings remain visible around the device.

The screen is a window onto the augmented scene

Augmented reality techniques

The diagrams show the path from the physical scene to the eye. Grey represents the real scene; cyan represents virtual content and its display path. Each illustration shows a different viewing arrangement.

Optical see-through: light from the real environment passes through a transparent optical combiner toward the eye. A projection system adds virtual imagery to that optical path. The physical world remains directly visible through the optics.

Video see-through: cameras capture the physical scene. Their images are combined with computer-generated content and the result is presented on a display. The eye sees the displayed camera view rather than looking through a transparent optical path.

Handheld display: a phone or tablet captures the scene with its camera and shows the augmented result on its screen. The user looks at this screen while the surrounding physical world remains visible around it. This is a device arrangement, usually using video see-through, rather than a third independent optical principle. Head-mounted displays can use optical or video see-through.

Augmented reality: examples

Optical see-through: HoloLens

Handheld display: ARKit

Augmented reality: examples

HoloLens illustrates optical see-through augmentation: the physical scene remains visible through the optics while virtual imagery is added. The design question is how the virtual content is positioned and updated relative to surfaces and the moving observer. Alignment, visibility and the correspondence between gesture and response determine whether the added content can serve the intended activity.

ARKit illustrates augmentation through a handheld camera view. The system estimates how the device moves and how virtual content relates to the filmed scene. The screen combines the camera image and rendered objects, while the surrounding physical space remains directly visible around the device. Maintaining registration as the camera moves is central to the illusion of an object located in the room.

Auditory stimulation

Make events audible and let the listener locate sound sources

Families of audio output

Headphones

Binaural rendering for the two ears

Loudspeakers

Physical sources and spatial reproduction arrays

3-D audio demonstration

Use headphones · enable sound

Haptic stimulation

Create sensations of contact, vibration and object resistance

Families of haptic output

Cutaneous feedback

Vibration, pressure and stimulation of the skin

Kinesthetic feedback

Forces and resistance applied to movement

Cutaneous feedback with contact

Vibrating controller: Joy-Con

Vibrating vest: bHaptics

Contactless cutaneous feedback

Air vortices: AIREAL

Ultrasound: Touchable Holography

Force-feedback arms

Stylus: PHANTOM Premium 1.5

Arm: Haption Virtuose 6D

Force feedback: examples

Cables: SPIDAR-8

Gloves: SenseGlove Nova 2

Force assistance driven by muscle activity

Proprioceptive stimulation

Engage the sense of the body’s position and movement

Families of movement interfaces

Active walking

Treadmills, moving surfaces and spherical devices

Assisted movement

Supports and mechanisms that guide the body

Moving walking surface

Spherical locomotion interface

Movable support surface

Vestibular stimulation

Create a sensation of acceleration or tilt

Families of vestibular stimulation

Physical movement

Platforms that accelerate or tilt the participant

Galvanic stimulation

Electrical interfaces investigated in research

Motion platform: flight simulator

Motion-platform and vestibular-stimulation research examples

Motion platforms: seat and hexapod

Seat: Motion Platform V3

Hexapod: PS-6TL-250

Robotic motion platform: DLR simulator

Galvanic vestibular stimulation

Olfactory stimulation

Present a smell associated with an object or environment

Families of olfactory interfaces

Ambient delivery

A scent is released into the environment

Local delivery

A device is worn or brought close to the nose

Olfactory output: aBio

Gustatory stimulation

Present tastes or alter the perceived flavour of food

Families of taste interfaces

Taste substances

Solutions or gels in contact with the tongue

Electrical modulation

Electrodes or controlled transport of substances

Multisensory approaches

Alter perceived flavour through vision and smell

Building a VR system

Course outline

Motion capture

Six degrees of freedom

Mechanical tracking

Telepresence Mobile Robot (1991)

SPIDAR-8

Electromagnetic tracking

Current tracking technologies

Inside-out and outside-in: principle

Inside-out and outside-in tracking

Sensor fusion

Sensor fusion: OKVIS

Tracking quality

Controllers

Hand tracking

Eye tracking

Body tracking

Sensory output

Visual stimulation

Families of visual displays

Head-mounted display: FlyVIZ

Immersive projection: a CAVE

Active stereo: shutter glasses

Passive stereo: polarisation

Anaglyph stereo

Autostereoscopic displays

Multi-user projection: C1x6

Augmented reality techniques

Optical see-through

Video see-through

Handheld display

Augmented reality: examples

Optical see-through: HoloLens

Handheld display: ARKit

Auditory stimulation

Families of audio output

3-D audio demonstration

Haptic stimulation

Families of haptic output

Cutaneous feedback with contact

Vibrating controller: Joy-Con

Vibrating vest: bHaptics

Contactless cutaneous feedback

Air vortices: AIREAL

Ultrasound: Touchable Holography

Force-feedback arms

Stylus: PHANTOM Premium 1.5

Arm: Haption Virtuose 6D

Force feedback: examples

Cables: SPIDAR-8

Gloves: SenseGlove Nova 2

Force assistance driven by muscle activity

Proprioceptive stimulation

Families of movement interfaces

Moving walking surface

Spherical locomotion interface

Movable support surface

Vestibular stimulation

Families of vestibular stimulation

Motion platform: flight simulator

Motion platforms: seat and hexapod

Seat: Motion Platform V3

Hexapod: PS-6TL-250

Robotic motion platform: DLR simulator

Galvanic vestibular stimulation

Olfactory stimulation

Families of olfactory interfaces

Olfactory output: aBio

Gustatory stimulation

Families of taste interfaces

Taste output: Norimaki Synthesizer

Meta Cookie: vision, smell and taste