Mixing realities for lunar exploration

Anyone who has tried virtual reality (VR) applications knows about its many possibilities: from immersive video games and 360-degree movies, to virtual museum tours, meetings, or journeys to space – all from the comfort of your home. 

ESA’s extended reality (XR) lab at the European Astronaut Centre goes one step further and uses virtual, augmented and mixed reality to train astronauts for missions to the International Space Station and beyond. This includes virtual tours of the Space Station to familiarise astronauts with its modules and learning how to operate a robotic arm.  

Lora-Line Faure applied her background in cognitive sciences to prepare training for the Moon. During her internship at the XR lab, she developed MIRAGE, or Mixed Interactive Rendering with Gesture recognition for Enhanced hand-tracked LUNA augmented experience.   

This application uses a hand-tracking system to visualise and interact with objects in mixed reality. Mixed reality allows users to interact with both digital and physical objects. Users wear a virtual reality headset and see a menu displayed on their hand, allowing them to place and interact with various objects, such as a lunar lander or rover projected directly in front of them. 

Landing on and exploring our Moon, as planned with the ESA-powered Artemis missions and the construction of the Gateway station in lunar orbit, will help us understand the formation of life on Earth and the Universe as a whole. Discovering resources like water or minerals on the Moon can also provide valuable insights for sustainable and long-distance space missions to Mars and beyond in the future. 

Currently, MIRAGE aims to be used on Earth for visualisation and training in the ESA-DLR LUNA analogue facility, which is currently being built next to ESA’s Astronaut Centre in Cologne, Germany.

LUNA will be a testbed for rovers and astronauts to determine the best systems, tools, and operations for working and living on the Moon. The facility recreates the lunar surface and will be filled with simulated Moon dust, called regolith, and can recreate lunar day and night cycles, as well as reduced gravity through a suspension system. The level of gravity on the Moon is about one sixth of Earth's. 

In the future, Lora-Line’s interactive system could also be applied to hand tracking for surface operations using mixed reality displays for astronauts on the Moon.  

Engineers and scientists could use it to visualise models or run simulations without the need for building real size models, making quick and low-cost adjustments possible.  

Astronauts could benefit from more immersive training and simulations, combining a real environment with virtual elements. It is also valuable for showing 3D models to help users get a better sense of proportion, as with the Argonaut lunar lander. 

For Lora-Line, a challenge of this new approach was to accurately track the real world with the headset and to introduce hand tracking to people unfamiliar with the technology.   

“I'm happy to have had the chance to learn, experiment, and develop such an innovative and stimulating application,” she reflected. For her, the internship was a full success, and she plans to continue the path of turning virtual concepts into reality.

Click here to access a video demonstration of MIRAGE.