Multimodality

AppleInsider reports on a set of patents for multimodal sensing (i.e. using two or more senses at the same time). Multimodal sensing has been a hot research topic in human-computer interaction for several years, based on the knowledge that human perception and cognition is fundamentally multimodal. If we want computers to respond more efficiently to human communication they will also have to use more than one modality in their sensing and communication. That said, I am not sure that everyone will be comfortable leaving the webcam on at all times to allow for computer vision techniques on everything happening in front of the screen (as the picture below depicts)… ...

Stefania Serafin just held a great lecture on multimodal perception and sonic interaction design at the SMC summer school. It is fascinating with the differences in spatiotemporal resolution between the different senses: Spatial resolution eye — highest: very high spatial acuity (foveal acuity ≈ 1 arcmin), excellent for fine detail. tactile — medium: fingertip resolution ≈ 1–2 mm, good for textures and small features. ear — lowest: coarse spatial localization (degree-level accuracy), poor for fine spatial detail. Temporal resolution ear — highest: excellent temporal sensitivity (sub-ms to ms scale), great for timing and rapid changes. tactile — medium: temporal resolution on the order of a few ms to tens of ms. eye — lowest: relatively slow temporal processing (temporal integration ~50–200 ms; critical flicker fusion ≈ 50–60 Hz). Stefania showed a number of examples of the application of this knowledge in interaction design, one example being the tactile floor they are building at McGill.