We describe the design of a system consisting of several state-of-the-art real-time audio and video processing components\r\nenabling multimodal stream manipulation (e.g., automatic online editing for multiparty videoconferencing applications) in open,\r\nunconstrained environments. The underlying algorithms are designed to allow multiple people to enter, interact, and leave the\r\nobservable scene with no constraints. They comprise continuous localisation of audio objects and its application for spatial audio\r\nobject coding, detection, and tracking of faces, estimation of head poses and visual focus of attention, detection and localisation\r\nof verbal and paralinguistic events, and the association and fusion of these different events. Combined all together, they represent\r\nmultimodal streams with audio objects and semantic video objects and provide semantic information for stream manipulation\r\nsystems (like a virtual director). Various experiments have been performed to evaluate the performance of the system.Theobtained\r\nresults demonstrate the effectiveness of the proposed design, the various algorithms, and the benefit of fusing different modalities\r\nin this scenario.
Loading....