Multimodal interaction techniques for astronaut training in virtual environments

This paper describes multimodal interaction techniques used in a virtual environment training prototype for astronaut assembly sequence training. Astronaut training takes place on Earth, which makes the simulation of zero gravity conditions necessary. Performing assembly tasks on orbit differs dramatically from ground operation in terms of object handling as well as user movement. The use of virtual environment techniques in astronaut training offers potentially significant cost savings and quicker access to the training situation compared to methods like parabolic flights. The usefulness of such a training application depends largely on the high enough resemblance of the operation to the real situation. In this respect the selected combination of interaction techniques plays a vital role. This paper presents a possible interaction technique battery for astronaut training that utilises two-handed interaction, speech and gesture recognition in a physically simulated virtual environment. User feedback is provided via head tracked stereo graphics and audio cues.