ARJ » sonification http://www.arj.no Alexander Refsum Jensenius Fri, 03 Feb 2012 12:51:10 +0000 en hourly 1 http://wordpress.org/?v=3.3.1 Sonification of motiongrams http://www.arj.no/2012/02/03/sonification/ http://www.arj.no/2012/02/03/sonification/#comments Fri, 03 Feb 2012 07:47:07 +0000 alexarje http://www.arj.no/?p=1760 A couple of days ago I presented the paper “Motion-sound Interaction Using Sonification based on Motiongrams” at the ACHI 2012 conference in Valencia, Spain. The paper is actually based on a Jamoma module that I developed more than a year ago, but due to other activities it took a while before I managed to write it up as a paper.

See below for the full paper and video examples.

The Paper

Abstract: The paper presents a method for sonification of human body motion based on motiongrams. Motiongrams show the spatiotemporal development of body motion by plotting average matrices of motion images over time. The resultant visual representation resembles spectrograms, and is treated as such by the new sonifyer module for Jamoma for Max, which turns motiongrams into sound by reading a part of the matrix and passing it on to an oscillator bank. The method is surprisingly simple, and has proven to be useful for analytical applications and in interactive music systems.

Full reference: A. R. Jensenius. Motion-sound interaction using sonification based on motiongrams. In ACHI 2012: The Fifth International Conference on Advances in Computer-Human Interactions, pages 170–175. IARIA, 2012.

@inproceedings{Jensenius:2012d,
    Author = {Jensenius, Alexander Refsum},
    Booktitle = {ACHI 2012: The Fifth International Conference on Advances in Computer-Human Interactions},
    Pages = {170--175},
    Publisher = {IARIA},
    Title = {Motion-sound Interaction Using Sonification based on Motiongrams},
    Year = {2012}}

Video examples

Video 1: A screencast demonstrating the jmod.sonifyer~ module.

Video 2: Examples of sonification of some basic movement patterns: up-down, sideways, diagonal and circular.

Video 3: One attempt at sonifying the two axes at the same time. Here both horizontal and vertical motiongrams are created from the same video recording, and the sonifications of the two motiongrams have been mapped to the left and right audio channel respectively.

Video 4: Examples of the importance of filtering and thresholding of the motion image for the final sounding result. The recordings were done at high-speed (200 fps) and played back at 25 fps.

Video 5: Sonification of a short violin improvisation (courtesy of Victoria Johnson).

Video 6: Sonification of a piece by a French-Canadian fiddler (courtesy of Erwin Schoonderwaldt).

Video 7: Sonification of free dance to music.

Video 8: Soniperforma: Performing with the sonifyer at Biermannsgården in Oslo on 18 December 2010. The performance was improvised and based on applying only video effects to change the sonic quality.

]]> http://www.arj.no/2012/02/03/sonification/feed/ 0 Sonification of motiongrams http://www.arj.no/2010/11/09/sonification-of-motiongrams/ http://www.arj.no/2010/11/09/sonification-of-motiongrams/#comments Tue, 09 Nov 2010 10:09:31 +0000 alexarje http://www.arj.no/?p=1248 I have made a new Jamoma module for sonification of motiongrams called jmod.sonifyer~. From a live video input, the program generates a motion image which is again transformed into a motiongram. This is then used as the source of the sound synthesis, and “read” as a spectrogram. The result is a sonification of the original motion, plus the visualisation in the motiongram.

See the demonstration video below:

The module is available from the Jamoma source repository, and will probably make it into an official release at some point.

]]> http://www.arj.no/2010/11/09/sonification-of-motiongrams/feed/ 2 Sonification of Traveling Landscapes http://www.arj.no/2008/05/15/sonification-of-traveling-landscapes/ http://www.arj.no/2008/05/15/sonification-of-traveling-landscapes/#comments Thu, 15 May 2008 08:22:04 +0000 alexarje http://www.arj.no/2008/05/15/sonification-of-traveling-landscapes/ I just heard a talk called “Real-Time Synaesthetic Sonification of Traveling Landscapes” (PDF) by Tim Pohle and Peter Knees from the Department of Computational Perception (great name!) in Linz. They have made an application creating music from a moving video camera. The implementation is based on grabbing a one pixel wide column from the video, plotting these columns and sonifying the image. Interestingly enough, the images they get out (see below) of this are very close to the motiongrams and videograms I have been working on.

Picture 1.png

]]> http://www.arj.no/2008/05/15/sonification-of-traveling-landscapes/feed/ 0