How can we convey expressive content to
users in a shared Mixed Reality environment?
What kind of techniques can we utilize to synthesize expressive gestures?
In order to answer these questions, research on expressive
content synthesis focuses on methods to generate expressive multimodal gestures.
Experiments concern possible uses of techniques coming from theatre and cinema
and aim at implementing output systems able to produce in the mixed reality
environment an expressive situation as required by the mapping strategies.
Naturally, research on expressive content synthesis takes into account the
communication of expressive content both in the particular output channels
(e.g., communication of expressive content through sound, music, movements,
visual media) as well as in a multimodal perspective i.e. synthesis of expressive
content deriving from combining several non-verbal communication channels.
On the side of expressive synthesis in audio research
Music performance. The score is given in a
MIDI-based format. The performance of the score can be controlled in
real-time using high-level gestural input control
modifications. The sound of a singer can be modified in real-time
using the GEM vocal processor hardware as well as using other modules
(time-frequency related) integrated in EyesWeb. The gestural input can be
mapped to parameters such as pitch of an added voice.
The position and movement of a sound in the room (3D) can be controlled by
processing. Control of musical and sonic parameters of pre-recorded
audio. A sound processing framework based on sinusoidal modeling provides
musically relevant sound transformation capabilities, in particular: time
stretching, pitch shifting, and timbre control, has been realized.
Gestural control for interaction with the sound processing engine
integrated in EyesWeb has been experimented.
On the visual side, the work consisted in the
development of a collection of EyesWeb blocks and patches for real-time
generation and processing of visual content depending on extracted
expressive audio and motion cues.
The project also
developed a 3D world consisting of an “underwater-world” containing several marine animals, for instance sharks,
dolphins and stingrays, in addition to other sea-elements as seaweed and
conches. The creatures are able to move around, and the world also contains
elements that make the environment more realistic, e.g., water-bubbles and
swaying seaweed. Plankton and other floating objects might be added to the
scene as well. Aspects of the environment (e.g., the density of water-bubbles, the number of
floating objects) and expression of the avatars/characters can be controlled
in real-time by parameters coming from analysis of gesture in music and human
movement. The underwater world has been employed for the interactive
game "Ghost in the Cave" (Stockholm, August 2003).
concrete output the research work produced a collection of software modules
for analysis and synthesis of expressive gestures, integrated or connected in the
MEGA System Environment. The research outputs have been employed in a
series of multimedia performances.