Making robot swarms legible with Augmented Reality

Wed, 20 Apr 2022 07:54:23 +0000

ANR-funded project (ANR-21-CE33-0006), coordinated by Jérémy Rivière, October 2021–October 2025.

At a glance

ARTUISIS asks a deceptively simple question: how can a person understand, monitor, and influence a swarm of autonomous robots without losing sight of its collective dynamics? The project brings together swarm robotics, tangible interaction, and Augmented Reality (AR) to make the local mechanisms behind a collective behaviour visible where they matter: around the swarm itself.

The scientific direction and the swarm-robotics focus were led by Jérémy Rivière. My contribution was to co-supervise Aymeric Hénard’s doctoral work and to focus on the AR visualisation strand: identifying which information a human operator needs, designing spatially localised visual cues, and evaluating whether they help people anticipate and prevent a loss of swarm cohesion.

From local rules to an intelligible collective

Robot swarms are decentralised systems made of many small, autonomous agents. Their collective behaviours—such as flocking, aggregation, expansion, coverage, or shape formation—emerge from local interactions rather than from a central controller. This makes swarms robust and scalable, but also difficult to read: the operator sees the global motion without necessarily seeing the forces, neighbourhood relations, or changes that produced it.

ARTUISIS addresses this gap through three complementary goals:

Explain self-organisation by relating visible collective behaviours to their underlying local mechanisms.
Support supervision by communicating the dynamics of the swarm in real time.
Enable informed action through intuitive tangible interaction with the swarm’s spatial organisation.

The project therefore pairs Jérémy Rivière’s work on robot-swarm self-organisation with a human-centred question: what should be made visible, in what form, and at what location, for an operator to build a useful mental model of a dynamic distributed system?

Augmented visualisation for swarm monitoring

The AR work explored situated, localised visualisations: graphical cues registered to robots and to their immediate spatial organisation instead of a separate, abstract control panel. The aim is not to display every data stream, but to reveal the information that is otherwise hard to infer from robot trajectories alone—for example, inter-robot links, local virtual forces, direction and velocity, the convex envelope of the group, and signals of an emerging fragmentation.

This approach turns the physical workspace into an explanatory view of the system. It can be used in simulation and with tracked physical robots, allowing an operator to inspect swarm structure while retaining awareness of the real environment. The visualisation research was driven by a perceptual evaluation loop:

establish what people can already perceive from a swarm’s motion;
identify critical situations where perception is insufficient, notably a pending fragmentation;
design visual cues that make the relevant mechanisms perceptually available; and
test whether those cues improve monitoring and support an appropriate intervention.

The videos below illustrate this principle. The overlays make relationships and local forces visible as the swarm changes its collective organisation.

Featured publication

IEEE Transactions on Visualization and Computer Graphics · 2025

The project’s main AR result evaluates visual cues that help an operator monitor swarm cohesion and anticipate fragmentation.

Aymeric Hénard, Etienne Peillard, Jérémy Rivière, Sébastien Kubicki, Gilles Coppin (2025). Towards Augmented Reality Support for Swarm Monitoring: Evaluating Visual Cues to Prevent Fragmentation. IEEE TVCG.

PDF Project DOI

Doctoral research: Aymeric Hénard

Aymeric Hénard completed a PhD in the ARTUISIS project from October 2021 to September 2024, co-supervised by Gilles Coppin, Jérémy Rivière, Sébastien Kubicki, and Étienne Peillard. His thesis, “Augmented Reality and Tangible Interaction to Supervise and Interact with a Swarm of Robots”, was defended on 5 December 2024.

The thesis established a coherent progression from the foundations of swarm self-organisation to operator support. It contributed a classification of spatial self-organisation methods, studied how observers perceive and anticipate swarm fragmentation, and developed and evaluated augmented visualisations intended to make the dynamics of a swarm more understandable. This work provides a basis for AR interfaces that assist a human operator without replacing the swarm’s decentralised autonomy.

Project team

ARTUISIS brought together complementary expertise in swarm robotics, human factors, tangible interaction, and Augmented Reality. The project was coordinated by Jérémy Rivière, with Aymeric Hénard’s doctoral research forming its main scientific thread.

Project coordinator

Jérémy Rivière

Scientific lead of ARTUISIS, with a focus on robot-swarm self-organisation: the local mechanisms that produce collective behaviours and the ways an operator can understand and influence them.

Core doctoral research · 2021–2024

Aymeric Hénard

Aymeric’s doctoral research formed the main scientific thread of ARTUISIS, connecting the swarm’s micro-mechanisms with the operator’s ability to perceive, understand, and supervise its collective behaviour.

classified the methods that produce spatial self-organisation in robot swarms;
characterised human perception and anticipation of swarm fragmentation; and
designed and evaluated localised visual cues for AR swarm monitoring.

Read the official thesis record

AR visualisation · doctoral co-supervision

Étienne Peillard

Contributed the situated-visualisation perspective: selecting, designing, and evaluating AR cues that make swarm dynamics perceptible to an operator.

Human factors · doctoral co-supervision

Gilles Coppin

Brought expertise in human–computer interaction and the human factors of supervising autonomous systems.

Tangible interaction · doctoral co-supervision

Sébastien Kubicki

Contributed expertise in natural human–system interaction and tangible interfaces for representing and acting on swarm spatiality.

Research engineeringJérémie DonjatProject development and technical support

Student contributions

Research internship · 2023

Nolwenn Paluet

Developed the AR prototype for immersive, localised swarm-data visualisation. Her work connected the headset to the swarm gateway and introduced cues such as robot direction vectors and the swarm’s convex envelope, in both simulated and motion-capture environments.

Research internship · 2022

Tristan Guichaoua

Set up the tracking and virtual-perception environment for experiments with a MONA robot swarm, providing the technical foundation for observing robot motion and testing behavioural algorithms.

Visualising self-organisation

The visual material follows the same thread as the research: first observe an emergent behaviour, then expose the local information that helps explain it.

Collective behaviours in simulation

AggregationCollective structure emerging from local attraction, alignment, and repulsion.

CoverageDistributed robots coordinating to occupy an area.

Shape formationLocal rules and environmental constraints producing a global configuration.

Augmented cues around the physical swarm

Flocking · virtual forcesMaking the local forces that organise collective motion visible.

Expansion · coloured linksShowing the swarm’s evolving neighbourhood structure.

Flocking · thresholded linksFiltering relations to focus attention on meaningful connections.

Expansion · thresholded linksHighlighting potential changes in swarm cohesion.

Visualisation principle

Science en Theizh outreach poster

Funding

ARTUISIS was funded by the French National Research Agency (ANR) under reference ANR-21-CE33-0006.

Robot swarms | Étienne Peillard