RTD focus of organization
Software engineering, speech processing, human-computer interaction
Description of RTD activities
The objective of the research is to enable multimodal human-computer interaction by developing a methodology and software architecture for fusion of multiple sensory inputs. The research is part of a current project at the FON University. The overall project combines multiple modalities of eye movement, speech, pen gestures, pointing, and graphic illustration to add redundancy to human-computer communication, thereby helping to disambiguate the user’s inputs and uncover the user’s intent. The research focuses on information fusion from multiple modalities into a unified command representation. The user’s intent is determined from this representation through parsing and semantic analysis and passed to the system for execution.
Fusion of sensory information can be accomplished at three levels: data, features or decisions (commands). The research will take the approach of fusing the information at the decision-level, so called “late fusion”, which is common in current multimodal human-computer interfaces. However, current approaches to multimodal fusion suffer from several limitations: (i) they are deterministic and do not take into account the probabilistic nature of individual mode data as well as the integrated representation; (ii) they assume that the individual modes function independently of each other; and (iii) they suffer from ambiguities on both – signal and meaning level. The proposed work aims to address these limitations. Bayesian networks will be used for knowledge representation and probabilistic reasoning in order to fuse the outputs from different modalities. The timing of various modalities, e.g., speech and gaze, is critical in resolving ambiguities when fusing information from different sources.
Experiences in international Programmes
- Participation in 3 Tempus projects;
- Participation in one FP6 project.
Organisation is/was involved in projects funded by the following RTD programmes
Inserted / Updated
2010-04-23 / 2010-04-26