Augmented Reality | Étienne Peillard

Lab — Smartphone augmented reality

Thu, 26 Feb 2026 00:00:00 +0000

Objective

Create an augmented reality application capable of:

detecting an image used as a marker,
displaying a 3D object attached to this marker,
detecting a plane in the environment,
defining a play area,
creating interactions between markers and the area.

Part 1 — Project setup

Create a Unity project.
Switch the platform to Android in Build Settings.

To access logs on the phone, install Android Logcat from the Package Manager.

Install AR Foundation.
Install ARCore XR Plugin (Android) or ARKit XR Plugin (iOS).
Open Project Settings → XR Plug-in Management and enable:

ARCore in the Android tab
ARKit in the iOS tab (if needed)

In Project Settings → Player → Android:

set Minimum API Level: Android 10 / API 29
in Graphics APIs, disable Vulkan and keep OpenGLES3 only

Add to the scene:

AR Session
XR Origin (with AR Camera)

Mobile rendering configuration

In the project, open: Assets/Settings/Mobile_Renderer
In the inspector: Add Renderer Feature → AR Background Renderer Feature

Verify that the application launches on the phone and that the camera feed is visible.

When launching the app on the phone for the first time, allow camera access when requested by the system.

Part 2 — Image detection

Create a new Reference Image Library asset.
Add a marker image.
Add an ARTrackedImageManager component to the XR Origin.
Assign the image library.
Create a prefab for the 3D object to display on the marker.
Assign this prefab to the ARTrackedImageManager.

You can use the markers provided for this lab. The ImageLibrary asset is available here and the markers are available here.

Part 3 — Plane detection (selecting a plane + play area)

Objective

detect horizontal/vertical planes using AR Foundation
allow the user to tap on a detected plane
place a play area (GameBoard) on this plane

3.1 Add required AR components

On the XR Origin object:

ARPlaneManager

Detects and maintains planes.
Assign a Plane Prefab to visualize planes.

Create a simple Plane Prefab

Create → Empty GameObject
Name the object: ARPlanePrefab
Add the components:

AR Plane
AR Plane Mesh Visualizer
Mesh Renderer
Mesh Filter

Create a simple material (e.g. Unlit / Color transparent).
Assign this material to the Mesh Renderer.
Drag the object into the Project folder to create a prefab.
Assign this prefab in ARPlaneManager → Plane Prefab.

ARRaycastManager

Used to cast a ray from the screen to detect planes.

Recommended setting:

Detection Mode → Horizontal

3.2 Create a play area prefab (GameBoard)

Create a simple prefab:

a Quad lying flat
a semi-transparent material
scale around 0.5 m
optional: BoxCollider

3.3 Implement tap-to-place (New Input System)

In this lab we use the New Input System (Unity Input System Package) to detect taps.

Prerequisites

Open Project Settings → Player
Set Active Input Handling → Input System Package (New)

Principle

Create a script BoardPlacement.cs. The script should:

enable EnhancedTouch (at startup)
retrieve the first active touch
filter the Began phase
perform an ARRaycast toward planes (PlaneWithinPolygon)
create or move the GameBoard at the hit pose

Useful functions / APIs:

EnhancedTouchSupport.Enable() / Disable()
Touch.activeTouches
touch.phase (New Input System)
touch.screenPosition
ARRaycastManager.Raycast(...)
TrackableType.PlaneWithinPolygon
Pose
Instantiate(...)
transform.SetPositionAndRotation(...)

Checkpoint

You should be able to:

see detected planes
tap a plane
place the play area

Part 4 — Interactions

4.1 Basic interaction

Implement an interaction between:

detected markers
the play area

Possible examples:

an object appears when the marker enters the area
the marker moves an object within the area
the marker orientation modifies a parameter

This section is intentionally open.

4.2 Resources for going further

The AR Foundation package provides several useful features to enrich your project.

Image tracking

ARTrackedImageManager
events: trackedImagesChanged
access to marker position

Plane detection

ARPlaneManager
access to detected planes

AR Raycast

ARRaycastManager.Raycast()
allows interaction with the real environment

AR Anchors

ARAnchorManager
allows objects to remain fixed in space

Light estimation

ARCameraManager
adapts virtual lighting to the real scene

Face / body tracking (depending on device)

Face Tracking
Body Tracking

Official documentation:

https://docs.unity3d.com/Packages/com.unity.xr.arfoundation

Part 5 — Scenario design

You now have all the technical elements needed to create a complete AR experience.

Invent a small application or mini-game using:

one or several markers
a zone
at least one interaction.

Possible extensions

multiple markers
interactions between markers
visual effects
AR user interface

Deliverables

Report (PDF) including:

description of the application
screenshots
technical explanations
Git link to the Unity project

Grading

Project setup — 2 pts
Image tracking — 2 pts
Plane detection and placement — 4 pts
Interactions — 4 pts
Scenario design — 5 pts
Technical quality — 2 pts
Submission — 1 pt

Lab — Projective augmented reality

Thu, 19 Feb 2026 00:00:00 +0000

Projective Augmented Reality

▶ Base Code

Objectives

This tutorial introduces the fundamentals of projective augmented reality (AR), from camera-based event detection to spatial augmentation using a projector.

The reference application is a tic-tac-toe game, but the architecture can be reused for any surface augmentation scenario (e.g., augmented board games, scientific visualization, research experiments).

You will learn how to:

Calibrate camera, projector, and physical surface
Compute transition (homography) matrices between spaces
Detect surface modifications
Project consistent augmentations in real time

System Overview

We work in three different spaces:

Camera image space
Projector image space
Physical board space

These spaces are not aligned. Calibration is required to compute the transformations between them.

The main goal is:

Convert a detected move (camera space) into board coordinates
Convert a board coordinate into projector coordinates

Installation

Make sure Python and pip are installed.

The tutorial has been tested with Python 3.12 and 3.13. Other versions may work, but adjustments might be necessary.

Install required dependencies:

pip install numpy==1.26.4
pip install opencv-python==4.10.0.84
pip install opencv-contrib-python==4.6.0.66

OpenCV: Computer vision library
Aruco: Marker generation and detection module (included in opencv-contrib)

Documentation: https://docs.opencv.org/4.x/index.html

Marker Generation

Run:

python generateMarkers.py

If correctly installed, four markers defining the playing area will be generated.

Print and place them on the board.

Coordinate Conventions

Matrix Representation (Images)

Images are NumPy matrices in row-major order:

First index → row (vertical, downward)
Second index → column (horizontal, rightward)
(0,0) is top-left

Point Representation

Points follow image convention:

Origin: top-left
X → right
Y → down

Coordinate system:

(0,0) - X →
 |
 |
 Y
 ↓

Be extremely careful when switching between matrix indexing and geometric coordinates.

Code Structure

Four main components:

Camera
Projector
CoreAR
TicTacToe

All missing parts are marked with:

### TODO

1️⃣ Initialization

Camera

Automatically searches for available camera
Exposure configurable
Must release using:

camera.release()

Complete all TODO sections.

Projector

Constructor: no modification required.

Implement:

drawBlack()

Understand how drawing works in projector space.

CoreAR

Constructor only stores references.

No modification required.

2️⃣ Calibration

Camera Calibration

Goal:

Detect the four ArUco markers
Match detected corners with board coordinates
Compute homography between board and camera

Complete all TODO sections in:

calibrateCamera()

Projector Calibration

Principle:

Project markers at known projector positions
Detect them with camera
Convert camera detections to board coordinates
Compute projector → board homography

Complete:

drawMarkers() in projector
calibrateProjector() in CoreAR

3️⃣ Drawing in Projector Space

Implement:

draw()
checkCalibration()

This allows:

Displaying the game board
Validating calibration visually

Test full calibration pipeline.

4️⃣ Game Logic

The ticTacToe class is already implemented.

You now focus on detecting moves.

5️⃣ Move Detection

Strategy:

Capture reference frame
Compare current frame with reference
Compute difference image
Apply threshold
Apply morphological opening
Extract blob position
Validate stability over time
Update reference frame

Complete all TODO sections in:

findMove()

Intermediate imshow() lines can be uncommented to debug.

Image Processing Pipeline

Frame subtraction
Thresholding
Morphological opening
Contour extraction

This isolates new moves placed on the board.

Improvements & Tuning

You may adjust:

Camera exposure
Projector brightness
Threshold value

Fine-tuning greatly impacts detection robustness.

What You Have Built

At the end of this tutorial, you have implemented:

Camera-projector calibration
Multi-space homography estimation
Real-time surface augmentation
Basic event detection

This architecture can be reused for:

Augmented board games
Interactive surfaces
Experimental setups
Scientific visualization

Deliverables

Completed code (all TODOs filled)
Functional calibration
Fully working augmented tic-tac-toe

Optional:

Detection improvements
Graphical enhancements
Additional gameplay features

Clarifications (Based on Student Questions)

OpenCV Installation with Conda

If pip is not available, use:

conda install -c conda-forge opencv

In most cases, Aruco is installed automatically with this command.

If not, try:

conda install -c conda-forge aruco

If ArUco is still missing, you may need to install pip and use the opencv-contrib-python package.

Camera → Projector Projection Matrix (R2P)

The projection matrix from the real board to the projector space (R2P) is:

Computed inside CoreAR
Assigned in main.py via:

proj.R2P = core.R2P

There is no encapsulation mechanism here. The attribute is dynamically added to the Projector instance.

Inside the Projector class, you simply use:

self.R2P

This works because draw() is only called after the attribute has been defined.

Calibration Functions and Transition Matrices

`calibrateCamera()`

Detects physical ArUco markers placed on the board
⚠ Do not place markers on a black background
Computes the transformation matrix:

R2C (Real → Camera)

`calibrateProjector()`

The projector displays virtual markers
The camera detects those projected markers
Matching is performed between:
- Known projected marker positions
- Detected marker positions in camera space
A Camera → Projector transformation is computed

From this, we deduce:

R2P (Real → Projector)

So the full transformation chain is:

Real → Camera → Projector

And finally:

R2P = (C2P) ∘ (R2C)

End of tutorial.

Lab — Marker tracking with OpenCV

Thu, 12 Feb 2026 00:00:00 +0000

Marker Tracking

▶ Base Code

Requirements:

pip install opencv-python

pip install numpy

Introduction

The goal of this lab session is to introduce you to several fundamental concepts in computer vision through a concrete application: detecting and localizing a planar marker in a video stream.

You will work on:

detecting feature points (interest points),
describing these points (local descriptors),
matching points detected in two images,
computing a homography (projective transformation),
augmenting a video stream based on marker localization.

By the end of the lab, you will have a program that, given:

a reference image (the marker),
a video stream (webcam or video file),

automatically detects the presence of the marker in the scene and masks the marker area by replacing it with a white polygon (to visually validate detection).

Application Principle

The overall idea of the system is the following:

Load the marker image (reference image).
Open a video stream (webcam or video file).
Detect a set of feature points in the marker image (FP_M).
Compute descriptors for these points (FD_M).
For each frame I_t of the video stream:
- detect a set of feature points (FP_t),
- compute descriptors (FD_t),
- match FD_M and FD_t to obtain correspondences,
- filter these correspondences to retain only “good matches”.
Estimate the geometric transformation relating the marker plane to the frame: a homography H_{M->t} robustly computed using RANSAC.
Use H_{M->t} to:
- project the 4 corners of the marker into image I_t,
- mask the detected region by filling it in white.

Important note: this lab performs localization via “re-detection + matching” at each frame. It is not a temporal tracking approach (e.g., KLT/optical flow), but rather a robust and simple method to implement.

Feature Detection

The objective of feature detection (or more generally “interest regions”) is to automatically select image elements that exhibit distinctive properties: corners, textured regions, high-contrast areas, etc.

A feature detector takes an image as input and outputs a set of pixel coordinates corresponding to points considered “interesting” by the algorithm.

Important: a detector provides positions (and sometimes scale/orientation) but not a descriptive signature.

Examples of detectors: Harris, FAST, Shi-Tomasi, ORB, AKAZE, SIFT, etc.

Resources (to read after the lab):

Feature Description

A feature descriptor aims to numerically characterize the local appearance around an interest point.

It takes as input:

an image,
a list of feature points,

and outputs a vector (or matrix) of descriptors, one per point.

These descriptors represent a local “fingerprint” that enables comparison of points detected in different images.

Desirable properties of a descriptor:

invariance (or robustness) to rotation,
invariance (or robustness) to scale changes,
robustness to photometric variations (lighting),
robustness to moderate geometric transformations.

Examples: ORB, AKAZE, BRISK, SIFT (detector + descriptor), etc.

Resources (to read after the lab):

Matching

Matching addresses the following question:

Given a feature in the marker image, how can we find the most similar feature in the current video frame?

The general idea is:

Define a distance or similarity measure between descriptors.
For each descriptor of the marker, find the closest descriptor in the frame.

Possible measures: L1, L2, Hamming (for binary descriptors), ratio test, etc.

Matching methods:

Brute force: compares all descriptors pairwise (simple, potentially costly).
FLANN: approximate nearest neighbors (faster for large datasets).

After matching, a filtering rule must be defined to retain only relevant correspondences. A classical approach (used in the provided base code) consists of using a threshold based on the minimum observed distance:

threshold = alpha * minDist

where alpha is a coefficient (typically between 3 and 20 depending on the method and expected quality).

Resources (to read after the lab):

Homography and Projection

Once correspondences between the marker (planar object) and the frame (image) have been established, we estimate the geometric transformation relating these two sets of points.

For a planar object, this transformation is a homography (3x3 matrix). It models perspective effects and relates two views of the same plane.

We estimate a matrix H from the good matches (after filtering). The estimation must be robust to outliers: we use RANSAC.

Resources (to read after the lab):

Implementation in OpenCV

The objective of the lab is to implement the previous steps in OpenCV starting from a provided code skeleton.

Provided Base Code

You are given the following files:

main.py
MyFeatureDetector.py
MyDescriptorExtractor.py
MyDescriptorMatcher.py

These files contain TODO sections guiding your implementation.

Important: the current base code loads two static images. You must adapt it to handle:

a reference image marker.jpg,
a video stream (webcam or file).

You will remove the final warped image generation part and replace it with an augmentation step (masking the marker) based on projected corners.

Questions

Question 1: Reading and Understanding the Provided Code

Read the provided code and identify:

where images are loaded,
where the detector, extractor, and matcher are instantiated,
where detected points and descriptors are stored,
how the list of best matches is built,
how the homography is computed.

Explain in a few lines the role of each file.

Question 2: Create a Feature Detector (ORB)

In MyFeatureDetector.py, complete the changeFeatureDetector function to create an ORB detector.

Objective: fill the myFeatureDetector attribute by calling the appropriate OpenCV constructor (e.g., cv.ORB_create(...)).

Question 3: Display Detected Points

In main.py, display detected feature points:

on the image marker.jpg,
on a video frame.

Hint: use the displayFeatures method and the existing TODO sections in the skeleton.

To do: add a screenshot showing results on the marker and on a frame.

Optional: add parameters to cv.ORB_create(...) (number of points, threshold, etc.) and comment on their impact.

Question 4: Instantiate a Descriptor Extractor (ORB)

In MyDescriptorExtractor.py, complete changeDescriptorExtractor to create an ORB descriptor extractor.

Question 5: Compute Descriptors

Complete computeDescriptors() in MyDescriptorExtractor.py and complete the corresponding calls in main.py to:

compute descriptors for the marker,
compute descriptors for a frame.

Verify that descriptors are properly computed (consistent dimensions, non-empty).

Question 6: Perform Matching

In MyDescriptorMatcher.py, complete the match method to retain only correspondences whose distance is below a threshold:

threshold = alpha * minDist

Selected matches must be stored in bestMatches.

Question 7: Display Matching Results

In main.py, call drawMatchingResults() and display the resulting image.

To do: illustrate this result with a screenshot in your report.

Question 8: Understand and Compute the Homography

From the best matches, build two lists of 2D points:

points in the marker image,
points in the video frame.

Then compute the homography using cv.findHomography(..., cv.RANSAC, epsilon).

Explain briefly what this computation does and why RANSAC is necessary.

Question 9: Augmentation — Mask the Marker

Once homography H is estimated, project the 4 marker corners into the frame.

Expected steps:

Define the 4 marker corners in its image coordinate system.
Project them using cv.perspectiveTransform.
Draw the contour (recommended for debugging).
Fill the projected polygon in white using cv.fillConvexPoly.

Condition: perform augmentation only if H is valid and if the number of inliers is sufficient (threshold to define and justify).

To do: provide a screenshot where the marker is effectively masked.

Question 10: Test on Other Sequences

Test your program with:

another video (or another webcam scene),
another marker (different reference image).

To do: illustrate at least one additional test and comment on robustness.

Question 11: Change Detector/Descriptor (Optional)

Try another detector/descriptor combination (e.g., AKAZE) and compare:

number of detected points,
matching stability,
number of inliers,
robustness to scale/lighting variations,
computational cost.

To do: illustrate and comment.

Deliverables

You must submit:

The completed code.
A short report (PDF or Markdown) containing:
- the requested screenshots,
- your parameter choices (alpha, inlier threshold, etc.),
- a robustness analysis (tests, failure cases).

Course — Tracking in mixed reality

Mon, 01 Jan 0001 00:00:00 +0000

Tracking in Mixed Reality

From Cameras to Robust Visual Tracking

Étienne Peillard – IMT Atlantique

🎯 Course Objectives

By the end of this lecture, you should be able to:

Explain what tracking means in Mixed Reality
Understand how cameras form digital images
Connect image processing to feature detection
Understand how features enable 6DoF tracking
See the link with the practical lab (feature detection for tracking)

PART A — Why Tracking in Mixed Reality?

What is Tracking in Mixed Reality?

Tracking = continuous estimation of pose:

Position (x, y, z)
Orientation (roll, pitch, yaw)

This applies to:

The camera/headset
The user’s body
Objects in the environment
Sometimes the entire scene (SLAM)

Why is Tracking Critical in MR?

Because MR requires spatial alignment between:

The real world (captured by sensors)
The virtual world (rendered by the engine)

If tracking is wrong:

Virtual objects drift
Registration breaks
The experience feels “fake” or unstable

The Registration Problem

Core question:

Given an image from a moving camera, how do we estimate its 6DoF pose in the real world?

Conceptual pipeline:

Three Families of Tracking in XR

Tracking and Latency

Good tracking must be:

Accurate
Low-latency
Robust to occlusion
Stable over time

Why? Because MR rendering depends on it in real time.

PART B — From Camera to Digital Image

Cameras as Sensors in XR

A camera is a physical sensor that:

Captures light from the environment
Converts it into a digital image
Introduces limitations that affect tracking

Key issues:

Discretization
Aliasing
Noise
Limited dynamic range

Discretization of Images

The real world is continuous.
A digital image is sampled on a grid of pixels.

This creates:

Loss of detail
Potential aliasing
Dependence on resolution

Interpolation

Aliasing and Moiré

If sampling is too coarse:

High-frequency patterns create artifacts (moiré)
Edges look jagged (aliasing)

This matters for tracking:

Artifacts can confuse feature detectors

Aliasing

Moiré

Demo: effect of downsampling on camera

Nyquist–Shannon Theorem

To properly sample a signal: Sampling frequency ≥ 2 × highest frequency in the signal

If violated → aliasing appears.

Color Quantization

Real light is continuous.
Digital images use a finite number of values:

Typically 8-bit per channel (0–255)
RGB representation

This limits precision in low-light conditions.

From RGB to Perception

Different color spaces exist:

RGB (device-oriented)
HSV (more aligned with human perception)
- Hue
- Saturation
- Value (brightness)

Useful in some tracking pipelines.

PART C — Projective Geometry (Core for Tracking)

Why Geometry Matters for Tracking

Tracking is fundamentally projective geometry.

We need to model:

How 3D points project onto a 2D image
How camera motion affects this projection

Intrinsic Parameters (K)

Describe the internal properties of the camera:

Focal length
Principal point
Pixel aspect ratio

Fixed for a calibrated camera.

Extrinsic Parameters (R, t)

Describe the pose of the camera in the world:

Rotation matrix R
Translation vector t

👉 This is what tracking estimates.

The Projection Model (Simplified)

The core equation:


u = K [R | t] X

Where:

X = 3D point in the world
u = 2D pixel in the image
K = intrinsics
[R|t] = camera pose (tracking result)

Intuition of the Projection

Transform 3D point from world to camera space (R, t)
Project onto image plane (K)
Obtain pixel coordinates (u)

This is the mathematical backbone of visual tracking.

PART D — Image Processing as a Precursor to Tracking

Why Image Processing for Tracking?

Three main goals:

Reduce noise → more stable features
Enhance structures → clearer edges/corners
Normalize images → robustness across lighting changes

Grayscale Conversion

Most feature detectors work on grayscale images.

Classic formula:

Y = 0.299R + 0.587G + 0.114B

Reduces complexity while preserving structure.

Histogram Equalization

Improves contrast:

Makes details more visible
Helps feature detection in low-contrast images

Useful in challenging lighting conditions.

Contrast and Brightness Adjustment

Convolution — The Key Tool

A sliding kernel is applied over the image.

Each pixel becomes a weighted sum of its neighbors.

This enables:

Blurring
Edge detection
Feature enhancement

Simple Blur

Gaussian Blur

Used to:

Reduce high-frequency noise
Smooth the image before detecting features

Important for robust tracking.

Gradients and Edges

Edges correspond to strong intensity changes.

We compute gradients using filters such as:

Sobel
Prewitt

Edges help locate meaningful structures.

Sobel Filter

$f’(i)=\frac{f_{i+1}-f_{i-1}}{2}\ \text{is equivalent to}\ [\tfrac12,\ 0,\ -\tfrac12]$

Laplacian (Second-Order Derivative)

Detects regions of rapid change:

Useful for feature detection
Highlights corners and fine details

The standard discrete Laplacian is written:

$\nabla^2 f(i,j)=f_{i+1,j}+f_{i-1,j}+f_{i,j+1}+f_{i,j-1}-4f_{i,j}$

This corresponds to the kernel:

From Image Processing to Features

At this stage, we have a cleaned and enhanced image.

Next step: 👉 Detect meaningful points for tracking.

PART E — Feature Detection for Visual Tracking

What is a Feature?

A feature is a distinctive image pattern that can be reliably detected and matched across frames.

Examples:

Corners
Blobs
Edges

What Makes a Good Feature?

A good feature should be:

Local (robust to occlusion)
Invariant (to translation, rotation, scale)
Robust (to noise, lighting changes)
Distinctive (easy to match)
Repeatable (found again in next frame)

Why Corners are Ideal

Corners are:

Stable across viewpoints
Well-localized in both x and y
Highly informative for motion estimation

Moravec Corner Detector (1977)

Idea:

Measure how much intensity changes when shifting a small window in different directions.

Average intensity variation for a small displacement ((u,v))

$E(u,v)=\sum_{x,y} w(x,y),(I(x+u,y+v)-I(x,y))^2$

(w) specifies the considered neighborhood (value 1 inside the window and 0 outside);
(I(x,y)) is the intensity at pixel ((x,y)).

Results

Limitation:

Not rotation invariant.

Harris Corner Detector (1988)

Improvement over Moravec:

Uses image gradients
More robust
Rotation invariant

Widely used in vision-based tracking.

Average intensity variation for a small displacement $(u,v)$

$E(x,y)=\sum_{u,v} w(u,v),|I(x+u,y+v)-I(x,y)|^2$

$w$ specifies the considered neighborhood (value 1 inside the window, 0 outside);
$I(x,y)$ is the intensity at pixel $(x,y)$.

Taylor expansion

$I(x+u,y+v)=I(x,y)+u\frac{\partial I}{\partial x}(x,y)+v\frac{\partial I}{\partial y}(x,y)+o(u^2,v^2)$

Neglecting $o(u^2,v^2)$:

$E(x,y)=\sum_{u,v} w(u,v)\left|u\frac{\partial I}{\partial x}(x,y)+v\frac{\partial I}{\partial y}(x,y)\right|^2$

Quadratic form

Average intensity variation for a small displacement ((u,v)):

For small displacements $(u,v)$

$E(x,y)=[u\ v]M[u\ v]^T$

$M$: symmetric, positive definite $\Rightarrow$ eigenvalue decomposition possible.

Structure of the second-moment matrix

$M=\begin{pmatrix}A,C;C,B\end{pmatrix}$

with:

$A=\left(\frac{\partial I}{\partial x}\right)^2 \otimes w ; B=\left(\frac{\partial I}{\partial y}\right)^2 \otimes w ; C=\left(\frac{\partial I}{\partial x}\frac{\partial I}{\partial y}\right)\otimes w$

$w$: Gaussian window (isotropic).

A corner is characterized by a large variation of $E$ in all directions of $(x,y)$.

$\Rightarrow$ Compute the eigenvalues of $M$.

Corner response

Instead of computing the eigenvalues, we can compute:

$\det(M)=AB-C^2=\lambda_1\lambda_2$

$\text{trace}(M)=A+B=\lambda_1+\lambda_2$

and define the response:

$R=\det(M)-k,\text{trace}(M)^2$

Values of $R$:

positive near corners,
negative near edges,
small in flat regions ($k=0.04$).

$\Rightarrow$ corners / interest points = local maxima of $R$.

Invariant by rotation: even after rotation, the matrix shape is unchanged, so the eigenvalues and the response $R$ are unchanged.

Not invariant by scale

Potential solution: compute Harris response at multiple scales (Harris–Laplace).

PART F — From Features to Tracking

Feature-Based Tracking Pipeline

Frame t:

Detect features
Describe features
Match with previous frame
Estimate motion (R, t)

Repeat continuously.

Feature Matching

Features are compared using descriptors.

Goal:

Find correspondences between frames

This enables motion estimation.

Example: SIFT (Scale-Invariant Feature Transform)

SIFT provides:

Scale invariance
Rotation invariance
Robust descriptors

Widely used in classical computer vision.

SIFT Pipeline

Build Gaussian pyramid (multi-scale representation)
Compute Difference of Gaussians (DoG)
Detect keypoints
Assign orientation
Compute descriptor from local gradients

Build Gaussian pyramid, compute DoG and detect keypoints

To obtain scale-independent descriptors, the image is resampled multiple times (building a “pyramid”).

The Gaussian difference is more efficient than a Laplacian calculation for each level.

Assign orientation and compute descriptor

The descriptor is a histogram of local gradients around the keypoint, weighted by a Gaussian window.

PART G — From 2D Features to 6DoF Tracking

The PnP Problem (Perspective-n-Point)

If we know:

3D world points
Their 2D projections in the image

We can solve for camera pose (R, t).

This is central to many AR systems.

Visual Odometry

If no 3D map is available:

Track motion between consecutive frames
Estimate relative movement of the camera

Used in SLAM systems.

SLAM in XR

Modern headsets use SLAM:

Simultaneous Localization and Mapping
Build a map of the environment
Track the headset within that map

Examples:

ARKit
ARCore
Meta Quest inside-out tracking

PART H — Tracking in Modern XR Systems

Inside-Out Tracking (Meta Quest)

Uses multiple cameras + IMU to:

Track headset position
Track controllers
Map environment

Fully vision-based.

Optical Tracking (Vicon/OptiTrack)

Uses external cameras to track reflective markers.

Very accurate but requires a dedicated setup.

Hybrid Tracking Systems

Combine:

Vision
IMU
Depth sensors

To improve robustness and reduce drift.