URL: http://www.multisense.org

Principal investigators: Andreas K. Engel (Hamburg) and Peter König (Osnabrück)

This project has been funded by an Advanced Grant of the European Research Council (ERC) between 2011 and 2016.

The integration of information across different sensory channels is a fundamental and complex problem that we have to solve in every moment of our daily life. Despite intense research in the past decades, we do not yet have a comprehensive view on the nature of unified multisensory experience and its underlying cognitive and physiological mechanisms.

This project has studied four key aspects of multisensory processing that are currently unresolved: (1) the role of dynamic binding in multisensory experience; (2) the role of action and sensorimotor contingencies in multisensory processing; (3) the modulation of multisensory experience by top-down factors; (4) the adaptivity of these three mechanisms in conditions of altered multisensory experience. To address these issues, we have applied psychophysiological, behavioral (e.g., eye tracking, sensory augmentation) and physiological (e.g., MEG, EEG, tACS) approaches in studies on human subjects and combined these with advanced signal analysis approaches.

Using these approaches, we have obtained neurophysiological evidence demonstrating that oscillatory brain signals in multiple frequency ranges are modulated by multisensory stimulation and are functionally relevant for implementing interactions across sensory systems. Importantly, we could also demonstrate that this involves changes in functional connectivity, mediated by phase coupling of neural oscillations. Our results show that factors influencing crossmodal interactions are manifold and operate in a stimulus-driven, bottom-up fashion, as well as via top-down control. Our data on sensorimotor interactions suggest that multisensory interaction and integration can be best understood within an embodied perspective, tying together sensory processing and behaviour in a unified framework. The results demonstrate a tight integration of sensory, cognitive and motor systems and the necessity to investigation of whole system in a ecologically valid context. In a set of studies on sensory augmentation we could demonstrate that newly acquired senses are integrated with the classic modalities. Furthermore, our investigations on adaptivity of multisensory processing show that developmental changes strongly shape the functional dynamics of the cortical networks. In blind participants, we could show that, due to the altered postnatal experience, functional connectivity within occipital cortex and between different cortical systems is profoundly altered.

During the implementation of the project, we advanced existing approaches and developed novel methodologies in several domains, including multisensory stimulus paradigms, sensory augmentation techniques, methods for neurostimulation and for removal of artefacts, as well as methods for analysis of BCI signals and of neural connectivity in EEG or MEG data. Furthermore, our work contributes to internet-based study methodology and to open-source data repositories. Dissemination of the results of the project was highly successful with a large number of publications, including original papers, review articles, talks and popular science presentations. Furthermore, the project has triggered the creating of two spin-off companies.

Taken together, this project has pursued a highly innovative and systematic approach, the results of which substantially augment our understanding of the nature and mechanisms of unified multisensory experience. Our project makes strong contributions to advancing a dynamic view on multisensory processing that includes contextual top-down modulation of multisensory processing, emphasises the relation to action, and considers dynamic functional connectivity as a key mechanism for multi sensory integration.