Mudra: A Unified Multimodal Interaction Framework

PDF BibTex

In recent years, multimodal interfaces have gained momentum as an alternative to traditional WIMP interaction styles. Existing multimodal fusion engines and frameworks range from low-level data stream-oriented approaches to high-level semantic inference-based solutions. However, there is a lack of multimodal interaction engines offering native fusion support across different levels of abstractions to fully exploit the power of multimodal interactions. We present Mudra, a unified multimodal interaction framework supporting the integrated processing of low-level data streams as well as high-level semantic inferences. Our solution is based on a central fact base in combination with a declarative rule-based language to derive new facts at different abstraction levels. Our innovative architecture for multimodal interaction encourages the use of software engineering principles such as modularisation and composition to support a growing set of input modalities as well as to enable the integration of existing or novel multimodal fusion engines.

ICMI2011

Kinect PowerPoint presentation

My kinect presentation movie is now on Youtube!


Go to gesture development to propose your own kinect gestures !

research/mudra.txt · Last modified: 2015/02/13 11:38 by lhoste
Trace: mudra