Augmented reality to help astronauts make sense of space

The ESA-designed Wearable Augmented Reality (WEAR) is a wearable computer system that incorporates a head-mounted display over one eye to superimpose 3D graphics and data onto its wearer’s field of view.

Controlled by voice for hands-free operation, WEAR includes onboard location and object identification to show astronauts precise information about what they are looking at, as well as providing step-by-step instructions to guide them through difficult, lengthy procedures.

“At the moment, International Space Station (ISS) crews still use paper instruction manuals for many operational and maintenance tasks,” explains Luis Arguello of ESA’s Modelling and Simulation Section, overseeing the WEAR project. “Obviously, it’s easier to perform a task while holding instructions in your hand. So we have developed a new type of user interface that is easier still, allowing astronauts to be guided precisely in their work without holding anything at all.”

From virtual to augmented reality

While the better-known concept of ‘virtual reality’ (VR) concerns immersive artificial worlds, ‘augmented reality’ (AR) involves blending the real world with simulated elements. The technology has already been employed in niche areas such as flight simulations and surgical training, but AR’s range of uses has increased as available computing power has grown.

“The WEAR concept started as research and development within ESA’s General Support Technology Programme, targeting commercial non-space applications such as architecture and maintenance as well as space activities,” explains Mr Arguello. “It’s the result of many years of VR/AR research. The system took a long time to conceive but was very fast to implement: once WEAR was accepted by ESA and NASA, we had to work fast to turn a design concept into flight-ready hardware."

Reflecting the pace of development, WEAR has been assembled from largely off-the-shelf components. Key hardware elements include a mobile computer connected to a headset with a head-mounted display, a pair of video cameras – for barcode reading and objection recognition – and an Inertial Measurement Unit (IMU).

The image-recognition system allows WEAR to know where its user is located, by checking his viewpoint against stored 3D information about the module, derived from a Columbus Computer Aided Design (CAD) model. The IMU works like the orientation tracking technology on an Apple iPhone, letting WEAR assess in which direction its wearer is moving and update his position and the image displayed accordingly.

On the software side, WEAR incorporates a 3D AR toolkit as well as speech recognition and synthesis technology, object recognition and tracking systems and commercial barcode reading technology. Reading the barcode allows the quick identification and retrieval of information of ISS items stored in the onboard Inventory Management System.

Technology demonstrator

The WEAR system, which was carried to the ISS on the Space Shuttle in August, is a technology demonstrator rather than a fully operational model, mainly due to schedule and budget constraints and some technical limitations in the computer hardware available on the ISS.

All equipment flown aboard the ISS must go through arduous safety and compatibility testing, and it would have taken too long to pass a faster ultramobile PC rather than use the IBM 31P Thinkpads already qualified and aboard the ISS. While state-of-the-art when first flown, these laptops are now more than five years old, though the good news is that more modern A61 Leonovo laptops are in the process of being introduced and should be available for future WEAR tests.

This demonstrator began its testing in ESA’s Columbus module on 19 September. An ISS Thinkpad has a battery life of only an hour with WEAR attached to it, so a task was chosen with that limitation in mind: opening up decking on Columbus to replace an internal filter. Frank De Winne was able to navigate by voice through each step of the procedure and visualise all the relevant information, diagrams and location information with WEAR, keeping his hands free for the maintenance work.

“I invited Frank in the early phases of the project to take part in reviews as a potential user in order to capture user requirements,” says Mr Arguello. “He became our main supporter and ended up presenting the idea to NASA.”

WEAR will be the third 3D visualisation system flown by ESA to ISS. Last year’s Automated Transfer Vehicle (ATV) delivered a pair of laptop-based training systems to ‘refresh’ astronauts in their knowledge of ATV systems.

This ESA project has Belgium-based Space Applications Services as its prime contractor, working with the Katholieke Universiteit Leuven as subcontractor in charge of the vision-based localisation software.

Meanwhile, on the ground, Space Applications is considering firefighting as a non-space application of WEAR. Within the space sector, it is being proposed to support operations inside ESA’s test facilities.

More information

Luis.Arguello @ esa.int